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1. Introduction 

Theoretical studies of neural network models of associative memory often involve the 
development of tools to study the dynamics of the network. In most simple models, the 
basic processing elements ( "neurons" ) are assumed to be two-state (Ising spin) variables, 
and the dynamics of the network is described by "update rules" that specify how the 
state of a neuron (spin) is governed by its net synaptic input (local field) due to the 
other neurons (spins) in the network. The interactions between different neurons are 
specified by the synaptic matrix obtained from the learning rule employed for the model 
[HE]. Dynamical studies shed light on the pattern recall process and its relation with the 
choice of the initial state, the learning rule, symmetry of the synaptic interaction matrix, 
etc. While methods of equilibrium statistical mechanics can be profitably used [TJ |2] to 
analyse the behaviour of neural network models with symmetric synaptic connections, 
dynamical techniques are the only tools available for the study of models with non- 
symmetric synaptic matrices. Since one is generally interested in the behaviour of large 
networks, a common strategy is to move away from the "microscopic" description of 
the dynamics of individual neurons and to derive a "macroscopic" description in terms 
of quantities (such as suitably defined "order parameters") that depend on the states 
of many neurons. The crucial question in this context is how the dynamical equations 
that describe the behaviour of these macroscopic quantities are to be derived from the 
microscopic dynamics of the neurons. 

The generating functional technique jSHUEHEJIIllH], which has been used extensively 
to study the dynamics of spin glasses and other disordered spin systems, provides an 
appropriate framework to accomplish this task. This technique has been used to study 
the synchronous dynamics of the Hopfield model [Oj and its asymmetric version [TTHlllj. 
Although the method allows, in principle, a calculation of all the properties of the 
network after an arbitrary number of time steps, it can, for all practical purposes, be 
used to follow the dynamics only for a few time steps because the the number of order 
parameters required in this description increases very quickly as the number of time 
steps is increased. This is not satisfactory because, in order to analyse the retrieval 
properties of a neural network, one needs a method that allows a study of the dynamics 
for long times. One can, of course, use numerical simulations for networks of finite size. 
However, extrapolating the results to the thermodynamic limit may be quite non-trivial 

Eisffeller and Opper have developed a numerical method for studying the parallel 
dynamics of the well-known Sherrington-Kirkpatrick (SK) [IHj model of spin glass 
with symmetric ^1] and asymmetric interactions JT2) in the thermodynamic limit. 
This method combines the generating functional method, which allows taking the 
thermodynamic limit exactly, and a Monte Carlo simulation of the resulting self- 
consistent single-spin stochastic dynamics. We have used this method to study the 
dynamics of pattern retrieval in a simple model of associative memory with one stored 
pattern. This model is essentially the same as the SK spin-glass model with an 
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interaction matrix that has a ferromagnetic component: the probability distribution of 
each element of the interaction matrix has a positive average. The ferromagnetic state 
in this model corresponds to the stored memory, and the random part of the interaction 
parameters represents the interference effects of the "other" memories in Hopfield-type 
models with a macroscopic number of stored patterns. The relative strength of the 
ferromagnetic part of the interactions plays the role of the "acquisition strength" of 
the stored pattern. Our study leads to a characterization of the retrieval behavior of 
this network as a function of this parameter. We also consider the effects of random 
asymmetry in the synaptic matrix on the retrieval performance of the model in the 
thermodynamic limit. The aim here is to shed light on the behavior of Hopfield-type 
models with random asymmetry in the synaptic connections. The main results of our 
study are as follows. 

For symmetric couplings, we find that the finite-time dynamical behaviour of the 
system exhibits a qualitative change at Jo = 1 where J is the acquisition strength of 
the stored pattern (relative strength of the ferromagnetic part of the interactions). This 
change may be described as a transition from spin-glass-like behaviour to ferromagnet- 
like behavior. In the ferromagnetic phase (Jo > 1), we find the existence of two types 
of phase-space structure for m > where the "magnetization" m is the overlap of 
the state of the system with the stored pattern. For large J , the system exhibits a 
simple phase-space structure where all initial states with m > flow to the attractor 
corresponding to the stored pattern. The phase-space structure for smaller values of 
Jo is complex, with many attractors with their basins of attraction. In the model with 
symmetric couplings, the process of retrieval of memory becomes very slow for small 
values of the initial overlap. The presence of random asymmetry in the couplings leads 
to an improvement in the retrieval performance of the network. The size of the basin of 
attraction of the stored pattern increases as an antisymmetric component is introduced 
in the synaptic matrix. The presence of synaptic asymmetry also decreases significantly 
the time the system takes to converge to the attractor corresponding to the stored 
memory. 

The paper is organized as follows. In Section|2]we introduce the model and its basic 
properties. The generating functional technique is used to construct a mean-field theory 
for the dynamics in Section El The results of the mean-field Monte Carlo simulations 
are presented and discussed in Section HI The last Section |S] contains a summary of the 
main results and a discussion of possible connections of these results with the behaviour 
of the Hopfield model with random synaptic asymmetry. 

2. The Model 

The model consists of N binary neurons (Ising spins) o~i = ±1, where every neuron <jj 
is connected to all other neurons <jj by couplings J if 
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where the first term represents Hebbian learning of the binary pattern with J being 
the acquisition strength for the pattern The second term is the coupling matrix of the 
SK model with random asymmetric interactions. As discussed in Ref. ^B], the synaptic 
interaction matrix given by Eq. ((TJ may be considered as a one-pattern analogue of 
the tabula non rasa scenario proposed by Toulouse, Dehaene, and Changeux [T2j . The 
couplings J^ K are taken to be independent Gaussian random variables for all i < j with 
distribution 

(4 K ) 2 



2it/N 



exp 



2/N 



1<J. 



(2) 



In addition, the symmetry of the coupling matrix is given by the average symmetry 
parameter r\: 

[Jfjf\=^, (3) 

where the brackets denote an average over the distributions of the couplings. The value 
7] = 1 denotes symmetric couplings whereas r] = — 1 corresponds to fully antisymmetric 
couplings. The case r) = corresponds to totally uncorrelated couplings. Couplings 
with these symmetry properties can be constructed via ^H] 
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where both J|. and Jff are independent Gaussian random variables for all i < j with 



distributions same as that given by Eq. (J2J), and J\ 
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This form of the synaptic interaction matrix is the same as that of the asymmetric SK 
model with ferromagnetic coupling Jo. In this paper we consider the zero-temperature 
(noise-free) synchronous dynamics of the model: 

<Ti(t + l) = 8ga(hi{t)) , i = l,...,N, (6) 

where the local field hi(t) acting on the spin <7j is given by 

hi(t) = ^JijCrjjt) , 



1+r] 



1/2 



1 — T] 



1/2 



(7) 



In the context of the synchronous dynamics of the asymmetric Hopfield model considered 
in Refs. [TTHITH]. the first term in the expression for the local field above is the signal term 
arising due to the pattern under retrieval. The second term mimics the noise arising from 
the interference of the other stored patterns (assuming the number of stored patterns to 
be a finite fraction of the number of neurons N). The last term which comes from the 
antisymmetric part of the synaptic interaction matrix is the same in both models. At 
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this point, it should be mentioned that in the limit of extreme dilution, the dynamics 
of the symmetrically diluted Hopfield model ^H] can be mapped onto the synchronous 
dynamics considered here [20J. Furthermore, as mentioned by Krauth et al. [21], the 
dynamics of our model for 77 = is equivalent to that of the asymmetrically diluted 
Hopfield model which was introduced by Derrida et al. [22]- 

For 77 ^ 1, the long-time dynamics of the model defined above for large but finite 
N is known to be rather complex [23J 121 123 I2HJ 1211 12EJ I2H] • In this paper, we will 
be concerned with the short-time dynamics of the model in the thermodynamic limit. 
The main objective here is to assess the retrieval performance of the network as an 
associative memory. To be specific, we shall study the initial value problem, where at 
time t — the spin configuration {<7j(0)} has a finite overlap m with the stored pattern. 
Since the stored pattern in the model is the ferromagnetic state, £j = 1 for all i, the 
overlap mo is nothing but the magnetization of the initial state. If the system evolves 
to a state whose overlap m with the stored pattern (magnetization) is sufficiently close 
to unity, then one speaks of successful retrieval of the pattern. Some of the issues that 
are of concern in this context are: 

(i) Retrieval quality, i.e., the closeness of the final state to the stored pattern. 

(ii) Basin of attraction, i.e., the volume of phase space occupied by initial states that 
converge to the attractor corresponding to the stored pattern. 

(iii) Convergence time, i.e., time taken by the network to converge to the attractor 
corresponding to the stored pattern. 

The dynamical mean-field theory described below allows us to address all these issues. 
3. Dynamical Mean-Field Theory 

The infinite range of interactions in our model makes it amenable to exact analysis 
using mean-field theory. A mean-field description involves an "effective field", that 
depends only on some macroscopic order parameters, instead of the actual fields hi(t) 
that depend explicitly on the states of all the spins. However, the formulation of such a 
theory is highly nontrivial because of the presence of quenched disorder in the synaptic 
interaction matrix. The effective field for disordered models like the one considered 
here turns out to be a rather complex time-dependent random process. The technique 
of dynamic generating functional provides an appropriate framework for constructing 
the random process for the effective field. This random process can then be studied 
numerically by generating stochastic spin trajectories in a Monte Carlo method. 

Let us consider the statistical properties of a finite, but large number Nt of spin 
trajectories of length tf, at the sites i — 1, . . . , Nt, in a system where the total number 
N of spins goes to infinity. These properties can be derived from the generating function 
(Z(\))j for the local fields hi(t), i = 1, . . . , N T ; t = 1, . . . , tf. 

(Z(l))j = (tt^ [([II Lh^t) 6 (a t {t + 1) Kit)) 5 (hi(t) °*(*) 

\ J i=i t=i I \ j^i 
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Here, (• • -)j denotes an average over the random couplings, Tr CT is the sum over all 2 Ntf 
possible combinations of the spin states <Ji(t) = ±1, and 9{x) is the unit step function. 
By construction only those "spin paths" <7j(t) consistent with the equations of motion 
© and © contribute to (Z(l))j. 

The calculation of (Z(l))j is a straightforward generalization of the derivation 
presented in Ref. jT3] for the case of Jq = 0. Introducing the integral representation of 
the (^-functions, we get 

(Z(\))j oc ( Tr a(t) [ Y[ {dhi(t)dhi(t) 6 (a^t + 1) Kit)) exp fe(t) (hi(t) 



i, t 
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where in the last exponential only the fields Zj(t) at the sites i = 1, . . . , Nt are different 
from zero. Overall constants in (Z(l))j which do not depend on the fields k(t) can always 
be recovered a posteriori, using the normalization relation Z(\ = 0) = (Z(l = 0))j = 1. 
As we will find out shortly, this relation is also useful in eliminating spurious solutions. 
As noted by de Domonicis [HO], since Z(l = 0) = 1 identically, one can compute directly 
(Z)j, the average of Z over the distribution of couplings, thus avoiding replicas. On 
averaging over the disorder {Jij} we get 

(Z(l))j oc Tr ffW j Ufa^dhiit) 8(^+1)^))} j^AkmhM+kWM) 

i,t 

X exp f ~Tn £ £ fa^«( s ) ff i(^ ff j( s ) +T]h i (t)a i (s)h j (s)a j (t) 



s,t 



X 



exp _i jEE hi{t)o-i{t) J 



(10) 



Introducing order parameters C(t, s), K(t, s) and m(t) 



m(t) 
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together with their conjugates C(t,s), K(t,s) and rh(t), respectively through the 
identities, 



t,s 



t,s 



j NdC(t, s) d ^'^ exp ^NC(t,s)C(t,s)-iC(t,s)J2< T j( t ) a i( s 
J i N dK(t, s) ^ exp (iN K(t, s) i K(t, s) - i K(t, s) H^j (*) 



Ndm(t) 



drh(t) 
2vr 



exp | iNrh(t) m{t) — irh(t) 0j(£) 



(12) 

,(13) 
(14) 



and neglecting terms of 0(1/N), we get 

(Z(l))j oc fj[[N dm{t) dm{t)} \ [ \n dC{t, s) dC{t, s) i NdK(t, s) dK{t, s) 



t.s 



exp J iiV ^ rh{t)m{t) + iiV ^ [c(t, s)C(t, s) + K{t, s)iK(t, 



+ ln [^(k m > ™> ^ K, K) 



(15) 



where the single-site partition function Z\ is given by is 
Z(k; m, m, C, C, K, K) oc Tr CTi ( t ) 



n{d^(t)d^(t)0(a,(t + l)^(t)) 

f 

x exp I i ^ ^ (t ) ^ (t ) + hi (t) hi (t) 

x exp < — iJ m{t)hj{t) — iy^rh(t)aj(t) 
L t t 

£ ( ^C(t, s)Ai(t)^(s) + iC7(t, s)o- i (t)a- i (s) 



s.i 



17/ 



+ s)/l<(t)<7i(s) + lK(t, s)/li(s)t7i(t) 



(16) 



At this stage the dynamical variables are decoupled with respect to their site index 
i. The exponent in Eq. (|15j) is of the form N F(m,rh,C,C,K,K). Therefore, in 
the limit N — > oo the integration over m(t), m(t), C(t,s), C(t,s), K(t,s), andK(t,s) 
can be performed using the saddle-point method. The stationary values of the order 
parameters are found from the following set of equations: 



M = £E <*<<«> 

i 



(17) 
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£(M) = -^£(W*(«)) , (19) 

i 



2 AT ^ \ * w 1V 7z, 

i 

K(t,s) = -^J2( k ^ a ^) 2i ■ ( 22 ) 

i 

In these equations, (■ • -}^. denotes an average with respect to the single-site partition 
function with = 0. Eqs. (J 17)) and (|19|) have only the trivial solutions, 

m(t) =0, (23) 
C(t,s) = 0, (24) 

as any other solution would violate the normalization Z(l = 0) = 1. 
From Eqs. (JHJ) and (J22J) we get 

K(t,s) = ^K(s,t). (25) 

As we will see below, K(t, s) is the average response of the magnetization at time t with 
respect to a small variation of the external field at time s. We are interested in the 
solutions that respect causality, i.e., 

K(t, s) = for s>t. (26) 

Once again using the normalization property of Z(l), we omit the single-site 
partition functions with l { = to get 

N T 



(Z(l))j cc n^(*) / Y[{dh i (t)dh l (t)Q(or t (t + l)h l (t)) 
i=i •* t 

cxpji^ (li{t) hi(t) + hi(t)hi(t) - ihi(t)J m{t)^ 

s,t s,t ) 

The generating functional ()2*7j) describes a system of noninteracting spins. It 
can be rewritten in a form where each spin is coupled to an effective field. In order to 
accomplish this we linearize the quadratic terms in hi(t) by introducing Gaussian random 
variables 4>i(t), with zero mean and covariance (cj>i{t)4>i(s)) ^ = C(t, s), independently for 
each site i. Using the identity 

ex p|-^E^(*)^( s ))^^*)^( s )| = ^exp j-i^<M*)^(t)Jy (28) 



Mean-field Monte Carlo Approach to the Dynamics of a One Pattern Model 9 

where (• • denotes an average over the time dependent Gaussian random variables 

(f>i(t), we get 



(Z(1))j oc J] (Tr CTi(t) f J] {dhi(t)dhi(t) 6 (o-i(t + 1) hi(t)) 
1=1 \ J t 

exp fij^ (k(t) hi(t) + hi(t)hi(t) - ihi(t)J m{tf 
-iY^Mt)Ut)-iV^K(t,8)^t)ai(8)\) . (29) 



s,t 



(30) 



On integrating over the auxiliary fields hi(t), we get to the result 
(Z(\))jOC Il(| Tr ^W / ^{dh l (t)Q(a ^ (t + l)h ^ (t))}exp^^J2h(t)h l (t) 

13 5 ^hi{t) - J m{t) - <f>i{t) -vJ2 K ^ s ^ ai ^ ) 

This representation of the generating function implies that the dynamics of the spin 
system given by Eq. (jUJ) is described by the uncorrelated system of stochastic dynamical 
equations: 

<Ti{t + l) = 8ga(hi(t)) , (31) 

with 

hi(t) = J m(t) + Mt)+vYl s ^ ai ^ ■ ( 32 ) 

s<t 

The first term in the "effective" local field is a simple disorder-free mean field term, the 
second term is a non-white Gaussian noise, while the third term represents a retarded 
self-interaction. 

The order parameters given by Eqs. (|TH|) . (|2T)|) . and (|2*2*|) can be rewritten in terms 
of the Gaussian averages: 

m{t) = (a(t)) , (33) 
C(t,s) = m)<K8))+=(*(t)v(s))+, (34) 

^-)=-i^(-M*)),= (^w) / (35) 

Eq. (|33jl clearly brings out the physical interpretation of K(t, s) as a response function. 
However, it would be highly inconvenient to use this relation to evaluate K(t, s) as 
it requires a calculation of the average of the partial derivative. Therefore, we use a 
discrete version of Novikov's theorem fH>\ l3~T] to express this quantity in terms of the 
correlation function (a(t)<p(s)) which is easier to estimate: 

t 

(a(t)<p(s)) = Y,K(t,r)C(r,s). (36) 

r=0 
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We note that Eq. (|36|) holds independently of the value of the asymmetry parameter 
77. On the other hand, a fluctuation dissipation theorem, which would enable us to 
express K(t, s) directly in terms of C(t, s), is not available for the asymmetric synaptic 
interaction matrix considered here. 

4. Results of Monte Carlo Simulation 

The single spin equations (J3"TJl and (J3"2j) can be used to calculate exact averages for 
iV — > 00 by expressing the spin variables as an explicit function of the Gaussian fields 
4>(t) and performing integrations weighted by the multivariate Gaussian measure. This 
integration is most conveniently performed by a Monte Carlo process, where a sequence 
of Gaussian random numbers with respect to the covariance C (t, s) is generated and a 
trajectory of spins a(t) is created via Eqs. (J3"T|) and (J32|) . The necessary average at each 
time step is estimated by summing over a large number Nt of trajectories. Nt should 
not be confused with N, the number of spins in the model, which tends to infinity. We 
closely follow the algorithm for the Monte Carlo simulation presented in Ref. |15| . We 
take o~ k (0) = ±1 with probabilities (l±mo)/2, respectively, for all k — 1, ... , Nt, where 
mo is the initial overlap with the stored pattern. In all our simulations we have taken 
Nt = 10 6 . Although, in most cases we have restricted the temporal range to tf < 200, 
occasionally we have gone beyond this range to bring out some qualitative features of 
the dynamics. 

4-1- Symmetric Couplings: rj = 1 

We first look at the stability of the stored pattern {1,1,. ..,1}. Accordingly, we start 
from <7 fc (0) = 1 for k = 1, . . . ,N T , i.e. m = 1, and evaluate m(t) for 100 time steps. 
Since the system would settle down, in general, to a limit cycle of length 2, we analyze 
the dynamics for even and odd times separately. We find that the results of simulations 
in both the cases can be fitted well by the function 



where the parameters a and are functions of Jo, the acquisition strength of the 
stored pattern, is the extrapolated value of the overlap for t even (or t odd ) — > 00. 
For estimating uncertainties in the values of the fitting parameters, m.00, const, and a, 
we take the uncertainty in the values of m(t) to be Am(t) = 10~ 3 , i.e., ~ 0(1 /y/N~r) 
at all time steps ^Hj- In order to have a cross-check on the results obtained from the 
mean-field Monte Carlo procedure, we did numerical simulation of Eqs. © and (JJJ) 
for J = 0.8 and m = 1. The calculations were done on finite samples of iV sites 
(25 < N < 5000) and the overlap m(t) was averaged over 100 to 2xl0 5 samples. In 
Fig. Q we plot moo(A^), the remanent overlap for the even time dynamics. By fitting 
moo(N) to the function 



m(t) 



moo + const x t 



(37) 



Woo (N) 



moo + const x iV 



-6 



(38) 
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we get — 0.36 ±0.05 which is in good agreement with m^ = 0.36 ±0.02 obtained by 
the mean-field Monte Carlo method described above. We show in Fig. El the remanent 




1000 2000 3000 4000 5000 
N 



Figure 1. Remanent overlap for the even time dynamics obtained by numerical 
simulation for Jo = 0.8, rj = 1, too = 1 and various values of the network size N. The 
full curve denotes the best fit of the results to the form of Eq. Ij38|l . 

overlap m^ and the exponent a of the power law decay of the even time dynamics in Eq. 
(137)1 as functions of Jo. The plots show clear evidence for a "transition" near Jo = 1: 
the rate of increase of the remanent overlap with increasing Jo is maximum near 
Jo = 1, and the exponent a has a minimum at the same value of Jo- This "transition" 
is from spin-glass-like to ferromagnet-like behaviour. The "spin-glass" phase for Jo < 1 
has a small value of the remanent overlap, arising due to the non-ergodic relaxation of 
the system through a complex energy landscape, which prevents it from reaching the 
equilibrium state corresponding to m = 0. The system gets frozen to a cycle of length 
two which can be characterized by the remanent overlaps m^ en and m^ G ld of the even and 
odd time dynamics, respectively. (Wherever we discuss the even time dynamics alone, 
we drop the superscript. Thus, m e ™ n and m.oo both refer to the remanent overlap for the 
even time dynamics). For Jo = 0, m™ n = 0.186 ± 0.001 whereas = (precisely, 
O(10 -3 ) which is the inherent level of errors involved in the calculation). Both of the 
remanent overlaps increase with Jo. Coming back to the even time dynamics we find in 
Fig. 2(b) that the relaxation becomes slower as Jo is increased in the spin-glass phase. 
On the other hand, in the "ferromagnetic" phase (Jo > 1), the system relaxes faster 
for higher value of Jo- The behavior is consistent with the physical intuition that Jo 
is the strength of the ferromagnetic coupling which opposes the decay of the system to 
a small value of moo in the spin glass phase, while helping the system to have a large 
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Figure 2. Results of fitting the mean- field Monte Carlo data for the evolution of the 
overlap, m{t), at even times to Eq. I|37[) for i] — 1 and tuq — 1. (a) Remanent overlap 
moo as a function of the acquisition strength Jq. (b) The exponent a of the power law 
as a function of Jo- 

above reflects a qualitative change in the short-time dynamical behavior of the system 
- it does not necessarily correspond to a phase transition in the thermodynamic sense. 
It is, however, interesting that the value of Jo where this change in the dynamics occurs 
is consistent with the phase diagram of the SK model with ferromagnetic interactions 
[32] , obtained by the replica theory, which shows a transition from the spin-glass phase 



Mean- field Monte Carlo Approach to the Dynamics of a One Pattern Model 13 

to a ferromagnetic phase (with replica symmetry broken) at J = 1. 

Strictly speaking, the stored pattern is not absolutely stable for any finite value of 
Jo: the remanent overlap is always less than unity. However, this does not prevent 
the network from performing as an associative memory. For sufficiently large values of 
Jo, we can have m.00 very close to unity (e.g., for J = 2.0, m^ = 0.942±0.001). m^ can 
be made as close to unity as we wish by increasing J . A similar scenario exists in the 
Hopfield model, where for extensive loading of memory, the stored patterns are not fixed 
points of the dynamics. In the Hopfield model, too, we have retrieval fixed points which 
can be made closer to the respective stored patterns by reducing the memory loading 
level of the network. What matters in both the models is the ratio of the strengths of 
the signal and the noise terms in the expressions for the local fields. This substantiates 
the analogy of J?- with the noise arising from stored patterns other than the one under 
retrieval in the Hopfield model. 

As mentioned earlier, the synchronous dynamics, in general, takes the system to 
a limit cycle of length 2. For the network to function as an associative memory, it 
is desirable that the values of the overlap to which the network settles down at even 
and odd times are not very different. In Fig. El we plot together m^ en and m^j d as 
functions of Jo- It is evident that for the values of the acquisition strength that are of 
interest (Jo > 1), the difference between the two remanent overlaps are very nominal. 
We, therefore, concentrate only on the even time dynamics hence after. 
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Figure 3. Remanent overlaps for the even-time and odd-time dynamics as functions 
of the acquisition strength Jo for T) = I and mo = 1. 

For the network to function as an associative memory, it also desirable that the 
attractor (limit cycle) corresponding to the stored pattern has a large basin of attraction, 
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i.e., a large number of initial configurations having a finite overlap with the stored 
pattern (i.e., with mo 7^ 0) should converge to this attractor. We, therefore, have 
studied the dynamics of the system with mo = 0.05, which is well below the remanent 
overlap for all values of J . We find that m(t) increases with time if J > 1, decreases 
with time if Jo < 1, and remains nearly constant for Jo = 1. This behavior, shown 
in Fig. |U also suggests a transition at Jo = 1. Even for the values of J > 1, two 
different kinds of behavior of m(t) are possible. For relatively smaller values of Jo, the 
value of moo depends on the initial overlap m , e.g., for J = 1.3 the values of moo for 
m = 0.05, 0.3, and 0.5 are quite different, as shown in Fig. indicating the presence 
of different "attractors" (other than the one corresponding to the stored pattern) with 
their basins of attraction. Thus we have a complex phase-space structure for such 
values of Jo- This is consistent with the known result jS2j that the zero-temperature 
ferromagnetic phase of the SK model is glassy with broken replica symmetry. On the 
other hand, for larger values of the acquisition strength, e.g. for J = 2, different initial 
values of m converge to the same moo, indicating a relatively simple structure of the 
phase space. In Fig. |H1 we show the evolution of m for initial overlaps ranging from 
mo = 0.005 to mo = 1 for Jo = 2. Since the initial overlap mo = 0.005 is already very 
close to the estimated value of Am = 0.001, it appears that all initial configurations with 
nonzero m converge to the same attractor that corresponds to the stored pattern. Such 
a large basin of attraction for sufficiently high values of the acquisition strength may be 
an artifact of the one pattern model. In the case of many stored patterns, the size of 
the basin of attraction of one of the patterns would get reduced. However, the simple 
one-parameter model brings out the essential feature that it is possible to tailor the size 
of the basin of attraction of a stored pattern by varying the corresponding acquisition 
strength. Note that our result for a simple phase-space structure for large values of J 
applies only to the subspace of states with finite values of m. It does not preclude the 
occurrence of a complex phase-space structure in the large subspace of states with zero 
magnetization. 

As mentioned earlier, the convergence time, which is the time taken by the network 
to reach the attractor corresponding to the stored pattern from an initial state in the 
basin of attraction of the attractor, is an important parameter in characterizing the 
performance of the network as an associative memory. From studies of spin-glass models, 
it is known that the dynamics for 77 = 1 may become very slow, especially for small 
values of the initial overlap mo jHBj- We also find evidence for slow dynamics in our 
calculations. For example, in Fig. 0we show m(t) for J = 1.5 and m = 0.05. Even at 
t = 350, m(t) does not show any sign of saturation. As there is no definite trend in the 
behavior of m(t), it is not possible to predict the value of moo and the corresponding 
time scale. 
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Figure 4. Evolution of the overlap m(t) at even times for different values of J for 
n = 1 and m = 0.05. 
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Figure 5. Evolution of the overlap m{t) at even times for different values of mo, the 
initial overlap with the stored pattern. The plots are for r\ = 1 and J = 1.3. 



4-2. Asymmetric Couplings: rj < 1 

As in the case of 77 = 1, we first look at the effect of asymmetry in the couplings (by 
lowering the value of rj) on the dynamics with the initial condition mo = 1. We find 
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Figure 7. Evolution of the overlap m(t) at even times for Jo = 1.5, r\ = 1 and 
too = 0.05. 



that the nature of relaxation changes from a pure power law to a combination of an 
exponential and a power law for rj < T] C1 . Accordingly, the results of simulations can be 
fitted very well by the function 

m(t) = moo + const x t" a exp(— t/r) . (39) 
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Thus for 7] < rj Cl , the overlap decays rapidly to with a finite relaxation time r. This 
behavior has also been reported in Ref. ^3] for J = 0. In that study, the remanent 
overlap was found to vanish at the same value of r], and the value of rj Cl was found 
to be 0.825. For J ^ 0, we find both quantitative and qualitative deviations in the 
behavior of m(t) from those reported in Ref. ^Hj. The value of rj Cl increases beyond 
0.825 as Jo is increased from zero. For example for J — 1.5, we have exponential 
relaxation for values of rj as high as 0.95. This is shown in Fig. El Moreover, 
vanishes for r] < r\ C2 < rj Cl . It is only at J = that r] C2 = r] Cl . The value of r\ C2 
decreases as J is increased. For example, for J = 0.8, r\ C2 ~ 0.65 whereas for J = 1.5 
r] C2 ~ —0.2. Furthermore, in the ferromagnetic phase, we find that over a considerable 
range of values of rj, there is very little variation in the values of m.oo, e.g. for J = 1.5, 
moo varies from 0.80 to 0.72 when r\ is reduced from 1 to 0. This feature of the network 
is highly desirable when the possibility of functional improvement is explored in the 
presence of asymmetry in the couplings. It ensures that the retrieval quality does not 
suffer significantly when r\ < 1. When Jo is sufficiently large, m^ always remains close 
to unity, e.g., for J = 2, m-oo varies 0.94 to 0.92 when rj is varied in its full range from 
1 to -1. 

Studies of models of spin glasses and neural networks suggest that the presence of 
asymmetry of an appropriate magnitude in the synaptic interaction matrix may result in 
the improvement of the performance of the network as an associative memory (see Ref. 
[To] for a detailed discussion on this aspect). It is expected that the asymmetry may 
destabilize some of the spurious attractors which do not correspond any stored pattern. 
If this happens, then the basin of attraction of the stored patterns would increase in 
size and the retrieval of memory would become faster. We find evidence of both of 
these effects. In Fig. |HJ we show the evolution of the overlap m(t) for various values of 
initial overlap ranging from mo = 0.01 to m = 1 for J = 1.3 and rj = 0.6. It can be 
easily seen that all these initial states converge to the same attractor with moo — 0.56. 
This signifies a considerable enhancement in the size of the basin of attraction when we 
compare this behaviour with that in Fig. EJ Note, however, that the retrieval quality 
has degraded in the model with synaptic asymmetry: the attractor corresponding the 
stored pattern has 68% overlap with the pattern for rj = 1 compared to the ~ 56% 
overlap for rj = 0.6. 

In Fig. El we plot m(t) for J = 1.5 and m = 0.1 for various values of the 
asymmetry parameter rj. It is very clear that the retrieval of memory becomes faster as 
the asymmetry in couplings is increased. By fitting m(t) with the function given in Eq. 
(I39|) . we find that the time constant r reduces from 55.56 to 4.61 when r\ is varied from 
0.95 to 0.6. At the same time, the value of the final overlap moo does not change much, 
indicating that the quality of retrieval is not substantially affected by the introduction 
of asymmetry in the synaptic interactions. 
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network with r\ = 0.6 and J = 1.3. 
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5. Conclusion and Discussions 

To summarize, we have studied the synchronous dynamics of a one-pattern model of 
associative memory using a mean field Monte Carlo method. Though simple, the model 
embodies sufficient richness to be useful in predicting the behavior of some of the more 
complicated models of associative memory such as the asymmetric Hopfield model. The 
two relevant parameters in the model are the acquisition strength J of the stored pattern 
and the symmetry parameter 77. For symmetric couplings (rj = 1), we find evidence for 
a transition at J = 1. For J < 1, we have a spin glass phase in which the retrieval 
overlap with the stored pattern is small, arising from a "remanence effect" (non-ergodic 
relaxation of the system through a complex energy landscape). On the other hand, for 
Jo > 1 we have a ferromagnetic phase where the retrieval overlap with the stored pattern 
increases rapidly with J and becomes very close to unity. Values of Jo > 1.3 would be 
required for a reasonable retrieval of the memory. Inside the ferromagnetic phase, we 
find the existence of two types of phase-space structure in the subspace of states having 
nonzero overlaps with the stored pattern. For example, for J — 2, we have a relatively 
simple phase-space structure where all initial states with a nonzero overlap with the 
stored pattern flow to the attractor corresponding to the stored pattern. In contrast, 
for J = 1.3 there are many attractors with their own basins of attraction. We also find 
that the process of retrieval becomes very slow for small values of mo- When random 
asymmetry is introduced in the couplings (77 < 1), we find that it results, in general, 
in better retrieval performance of the network by enhancing the size of the basin of 
attraction of the stored pattern, as well as by making the recall of memory significantly 
faster. 

How do the results described above compare with those obtained for the Hopfield 
model with random asymmetric interactions fSJEII? Numerical simulations in Ref. JH] 
have shown that the presence of asymmetry in the synaptic interaction matrix makes 
the convergence to spurious attractors slower. On the other hand, the convergence 
time for correct retrieval is only marginally increased in the presence of asymmetry. 
Thus, asymmetry in the synaptic interaction matrix enhances the performance of the 
Hopfield net as an associative memory by providing a way of discriminating between 
spurious and retrieval attractors by looking at the dynamics of the network. In the 
model studied here, the improvement in performance occurs in a more direct manner: 
the retrieval becomes faster in the presence of asymmetry in the synaptic interaction 
matrix. Furthermore, the simulation results of Ref. fH] show that the introduction of 
asymmetry in the synaptic interaction matrix does not cause any enhancement of the 
typical size of the basins of attraction of stored patterns in the Hopfield model. Here 
we do find an enhancement of the size of the basin of attraction of the stored pattern. 
However, this occurs for a very restricted range of values of the parameter Jo. These 
results may be indicative of the enlargement of the basin of attraction being a model 
specific feature. Confirmation of some of these issues by extending the method used 
here to the asymmetric Hopfield model and to other models using different learning 
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rules would be most interesting. A somewhat straightforward generalization of the 

technique for the case of two stored patterns with different acquisition strengths would 

provide useful insights into how the storage of other patterns, e.g., in the Hopfield model, 

would modify the results described above. Furthermore, this method may also be useful 

in analysing the effect of asymmetry in the synaptic interaction matrix of models of 

short-term memory ^3 HH] . 
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